update test workflow to use src layout

use pytest to manage docker container startup for tests

Chore/dev env fixes

 into impr/modernize-pre-commit

Impr/modernize pre commit

switch settings to use pydantic-settings

…tings-management-adsuG

- Remove ConfigWrapper class and backward compatibility layer
- Use direct pydantic BaseSettings with typed nested models
- Change context manager from config(...) to config.override(...)
- Add validate_assignment=True for runtime type checking
- Improve store spec validation with clear error messages
- Update all tests to use new API
- Preserve dict-style access via __getitem__/__setitem__ for convenience

Config file search:
- Search for datajoint.json recursively up from cwd
- Stop at .git/.hg boundaries or filesystem root
- Warn if no config file found (instead of silently using defaults)

Secrets management:
- Add .secrets/ directory support (next to datajoint.json)
- Support /run/secrets/datajoint/ for Docker/Kubernetes
- Use SecretStr for password and aws_secret_access_key
- Secrets masked in repr/logs, excluded from save()
- Dict access automatically unwraps SecretStr for compatibility

Breaking changes:
- Config file renamed from dj_local_conf.json to datajoint.json
- No more ~/.datajoint_config.json (project-only config)
- Secrets should be in env vars or .secrets/ directory

Add convenient type aliases that map to MySQL types:
- float32 -> float
- float64 -> double
- int32 -> int
- uint32 -> int unsigned
- int16 -> smallint
- uint16 -> smallint unsigned
- int8 -> tinyint
- uint8 -> tinyint unsigned

These aliases follow the same pattern as UUID, storing the
original type in the column comment for round-tripping.

- int64 -> bigint
- uint64 -> bigint unsigned

Add comprehensive tests for the new type aliases feature:
- Pattern matching tests for all 10 type aliases
- MySQL type mapping verification
- Table creation with type aliases
- Insert and fetch operations
- Primary key usage with type aliases
- Nullable column support

Co-Authored-By: Claude Opus 4.5 <[email protected]>

Co-Authored-By: Claude Opus 4.5 <[email protected]>

Co-Authored-By: Claude Opus 4.5 <[email protected]>

Documentation is now consolidated in datajoint-docs repository.

Changes:
- Delete docs/ folder (legacy MkDocs infrastructure)
- Create ARCHITECTURE.md with transpiler design docs
- Update README.md links to point to docs.datajoint.com

The Developer Guide remains in README.md. Internal architecture
documentation for contributors is now in ARCHITECTURE.md.

Co-Authored-By: Claude Opus 4.5 <[email protected]>

Transpilation documentation moved to datajoint-docs query-algebra spec.
Developer docs now consolidated in README.md.

Co-Authored-By: Claude Opus 4.5 <[email protected]>

- Rename CHANGELOG.md to CHANGELOG-archive.md with redirect to GitHub Releases
- Add "Writing Release Notes" section to RELEASE_MEMO.md:
  - Categories (BREAKING, Added, Changed, Deprecated, Fixed, Security)
  - Format template with examples
  - Guidelines for good release notes
  - PR label mapping for release drafter

Co-Authored-By: Claude Opus 4.5 <[email protected]>

- Slim README.md to essentials (intro, badges, install, links)
- Create CONTRIBUTING.md with:
  - Development setup (pixi and pip)
  - Test running instructions
  - Pre-commit hooks
  - Environment variables
  - Condensed docstring style guide
- Delete DOCSTRING_STYLE.md (merged into CONTRIBUTING.md)

README: 218 → 82 lines
All detailed docs now at docs.datajoint.com

Co-Authored-By: Claude Opus 4.5 <[email protected]>

- Add .github/DISCUSSION_TEMPLATE/rfc.yml for enhancement proposals
- Fix table header alignment (center instead of right)
- Fix excessive padding in table headers by removing p tag margins

Co-Authored-By: Claude Opus 4.5 <[email protected]>

- Raw blobs (no codec) now show "bytes"
- Raw json (no codec) shows "json"
- Codec fields show "<codec_name>" (e.g., <blob>, <hash>, <object>)
- HTML output properly escapes angle brackets for browser display
- Improves clarity when viewing table contents

Example output:
  *id    raw_blob  blob_data  json_data
  1      bytes     <blob>     json

Co-Authored-By: Claude Opus 4.5 <[email protected]>

Add migrate_external() and migrate_filepath() to datajoint.migrate module
for safe migration of 0.x external storage columns to 2.0 JSON format.

Migration strategy:
1. Add new <column>_v2 columns with JSON type
2. Copy and convert data from old columns
3. User verifies data accessible via DataJoint 2.0
4. Finalize: rename columns (old → _v1, new → original)

This allows 0.x and 2.0 to coexist during migration and provides
rollback capability if issues are discovered.

Functions:
- migrate_external(schema, dry_run=True, finalize=False)
- migrate_filepath(schema, dry_run=True, finalize=False)
- _find_external_columns(schema) - detect 0.x external columns
- _find_filepath_columns(schema) - detect 0.x filepath columns

Co-Authored-By: Claude Opus 4.5 <[email protected]>

Implement the `<npy@>` codec for schema-addressed numpy array storage:

- Add SchemaCodec base class for path-addressed storage codecs
- Add NpyRef class for lazy array references with metadata
- Add NpyCodec using .npy format with shape/dtype inspection
- Refactor ObjectCodec to inherit from SchemaCodec
- Rename is_external to is_store throughout codebase
- Export SchemaCodec and NpyRef from public API
- Bump version to 2.0.0a17

Key features:
- Lazy loading: inspect shape/dtype without downloading
- NumPy integration via __array__ protocol
- Safe bulk fetch: returns NpyRef objects, not arrays
- Schema-addressed paths: {schema}/{table}/{pk}/{attr}.npy

Co-Authored-By: Claude Opus 4.5 <[email protected]>

The SchemaCodec (used by NpyCodec and ObjectCodec) needs _schema,
_table, _field, and primary key values to construct schema-addressed
storage paths. Previously, key=None was passed, resulting in
"unknown/unknown" paths.

Now builds proper context dict from table metadata and row values,
enabling navigable paths like:
  {schema}/{table}/objects/{pk_path}/{attribute}.npy

Co-Authored-By: Claude Opus 4.5 <[email protected]>

…to feature/npy-codec

Merge PR #1330 (blob preview display) into feature/npy-codec.
Bump version from 2.0.0a17 to 2.0.0a18.

Co-Authored-By: Claude Opus 4.5 <[email protected]>

Address reviewer feedback from PR #1330: attr should never be None
since field_name comes from heading.names. Raising an error surfaces
bugs immediately rather than silently returning a misleading placeholder.

Co-Authored-By: Claude Opus 4.5 <[email protected]>

Show codec names in table preview instead of =BLOB=

Support memory-mapped loading for large arrays:
- Local filesystem stores: mmap directly, no download
- Remote stores: download to cache, then mmap

Co-Authored-By: Claude Opus 4.5 <[email protected]>

…orage

Major changes to hash-addressed storage model:
- Rename content_registry.py → hash_registry.py for clarity
- Always store full path in metadata (protects against config changes)
- Use stored path directly for retrieval (no path regeneration)
- Add delete_path() as primary function, deprecate delete_hash()
- Add get_size() as primary function, deprecate get_hash_size()
- Update gc.py to work with paths instead of hashes
- Update builtin_codecs.py HashCodec to use new API

This design enables seamless migration from v0.14:
- Legacy data keeps old paths in metadata
- New data uses new path structure
- GC compares stored paths against filesystem

Co-Authored-By: Claude Opus 4.5 <[email protected]>

- Remove uuid_from_buffer from hash.py (dead code)
- connection.py now uses hashlib.md5().hexdigest() directly
- Update test_hash.py to test key_hash instead

Co-Authored-By: Claude Opus 4.5 <[email protected]>

Remove dead code that was only tested but never used in production:
- hash_exists (gc uses set operations on paths)
- delete_hash (gc uses delete_path directly)
- get_size (gc collects sizes during walk)
- get_hash_size (wrapper for get_size)

Remaining API: compute_hash, build_hash_path, get_store_backend,
get_store_subfolding, put_hash, get_hash, delete_path

Co-Authored-By: Claude Opus 4.5 <[email protected]>

feat: Add NpyCodec for lazy-loading numpy arrays